Basics of probability and distributions
Random variables & probability
Probability is the expression of belief in some future outcome
A random variable can take on different values with different probabilities
The sample space of a random variable is the universe of all possible values
The sample space can be represented by a
- probability distribution (for discrete variables)
- probability density function (PDF - for continuous variables)
- algebra and calculus are used for each respectively
- the probabilities of an entire sample space always sum to 1.0
- There are many families or forms of distributions or PDFs
- depends on the nature of the dynamical system they represent
- the exact instantiation of the form depends on their parameter values
- we are often interested in statistics in estimating parameters
Bernoulli distribution
\[Pr(X=\text{Head}) = \frac{1}{2} = 0.5 = p \]
\[Pr(X=\text{Tails}) = \frac{1}{2} = 0.5 = 1 - p \]
- If the coin isn’t fair then \(p \neq 0.5\)
- However, the probabilities still sum to 1
\[ p + (1-p) = 1 \]
- Same is true for other binary possibilities
- success or failure
- yes or no answers
- choosing an allele from a population based upon allele frequences
Probability rules
- Flip a coin twice
- Represent the first flip as ‘X’ and the second flip as ‘Y’
- First, pretend you determine the probability in advance of flipping both coins
\[ Pr(\text{X=H and Y=H}) = p*p = p^2 \] \[ Pr(\text{X=H and Y=T}) = p*p = p^2 \] \[ Pr(\text{X=T and Y=H}) = p*p = p^2 \] \[ Pr(\text{X=T and Y=T}) = p*p = p^2 \]
Probability rules
- Now determine the probability if the
H and T can occur in any order
\[ \text{Pr(H and T) =} \] \[ \text{Pr(X=H and Y=T) or Pr(X=T and Y=H)} = \] \[ (p*p) + (p*p) = 2p^{2} \]
- These are the ‘and’ and ‘or’ rules of probability
- ‘and’ means multiply the probabilities
- ‘or’ means sum the probabilities
- most probability distributions can be built up from these simple rules
Joint and conditional probability
Joint probability
\[Pr(X,Y) = Pr(X) * Pr(Y)\]
- Note that this is true for two independent events
- However, for two non-independent events we also have to take into account their covariance
Joint and conditional probability
Conditional probability
- For two independent variables
\[Pr(Y|X) = Pr(Y)\text{ and }Pr(X|Y) = Pr(X)\]
- For two non-independent variables
\[Pr(Y|X) \neq Pr(Y)\text{ and }Pr(X|Y) \neq Pr(X)\]
- Variables that are non-independent have a shared variance, which is also known as the covariance
- Covariance standardized to a mean of zero and a unit standard deviation is correlation
Binomial Distribution
\[\large f(k) = {n \choose k} p^{k} (1-p)^{n-k}\]
n is the total number of trialsk is the number of successesp is the probability of successq is the probability of not success- For binomial as with the Bernoulli
p = 1-q
Binomial Probability Distribution
Binomial Probability Distribution
- Note that the binomial function incorporates both the ‘and’ and ‘or’ rules of probability
- This part is the probability of each outcome (multiplication)
\[\large p^{k} (1-p)^{n-k}\]
This part (called the binomial coefficient) is the number of different ways each combination of outcomes can be achieved (summation)
\[\large {n \choose k}\]
Together they equal the probability of a specified number of successes
\[\large f(k) = {n \choose k} p^{k} (1-p)^{n-k}\]
Poisson Probability Distribution
Poisson Probability Distribution
- For example, you can examine 100 plots of land
- count the number of snails in each plot
- what is the probability of observing a plot with ‘r’ snails is
Pr(Y=r) is the probability that the number of occurrences of an event y equals a count r in the total number of trials
\[Pr(Y=r) = \frac{e^{-\mu}\mu^r}{r!}\]
- Note that this is a single parameter function because \(\mu = \sigma^2\), and the two together are often just represented by \(\lambda\)
\[Pr(y=r) = \frac{e^{-\lambda}\lambda^r}{r!}\]
- This means that for a variable that is truly Poisson distributed:
- the mean and variance should be equal to one another, a hypothesis that you can test
- variables that are approximately Poisson distributed but have a larger variance than mean are often called ‘overdispersed’
Poisson Probability Distribution | gene length by bins of 500 nucleotides
Poisson Probability Distribution | increasing parameter values of \(\lambda\)
Log-normal PDF | Continuous version of Poisson (-ish)
Binomial to Normal | Categorical to continuous
The Normal (aka Gaussian) | Probability Density Function (PDF)
Normal PDF
Normal PDF | A function of two parameters (\(\mu\) and \(\sigma\))
where \[\large \pi \approx 3.14159\]
\[\large \epsilon \approx 2.71828\]
To write that a variable (v) is distributed as a normal distribution with mean \(\mu\) and variance \(\sigma^2\), we write the following:
\[\large v \sim \mathcal{N} (\mu,\sigma^2)\]
Normal PDF | estimates of mean and variance
Estimate of the mean from a single sample
\[\Large \bar{x} = \frac{1}{n}\sum_{i=1}^{n}{x_i} \]
Estimate of the variance from a single sample
\[\Large s^2 = \frac{1}{n-1}\sum_{i=1}^{n}{(x_i - \bar{x})^2} \]
The standard deviation is the square root of the variance
\[\Large s = \sqrt{s^2} \]
Normal PDF
Why is the Normal special in biology?
Why is the Normal special in biology?
Why is the Normal special in biology?
Parent-offspring resemblance
Genetic model of complex traits
Distribution of \(F_2\) genotypes | really just binomial sampling
Why else is the Normal special?
- The normal distribution is immensely useful because of the central limit theorem
- The central limit theorem states that, under mild conditions, the mean of many random variables independently drawn from the same distribution is distributed approximately normally, irrespective of the form of the original distribution
- One can think of numerous situations, such as when multiple genes contribute to a phenotype, that many factors contribute to a biological process
- In addition, whenever there is variance introduced by stochastic factors or sampling error, the central limit theorem holds as well
- Thus, normal distributions occur throughout biology and biostatistics
z-scores of normal variables | Mean centering and ranging
- Often we want to make variables more comparable to one another
- For example, consider measuring the leg length of mice and of elephants
- Which animal has longer legs in absolute terms?
- Which has longer legs on average proportional to their body size? Which has more variation proportional to their body size?
- A qood way to answer this last question is to use ‘z-scores’, which are standardized to a mean of 0 and a s.d. of 1
- We can modify any normal distribution to have a mean of 0 and a standard deviation of 1
- Another term for this is the standard normal distribution
\[\huge z_i = \frac{(x_i - \bar{x})}{s}\] ## Parameter Estimation | Ordinary Least Squares (OLS)
- Algorithmic approach to parameter estimations
- One of the oldest and best developed statistical approaches
- Used extensively in linear models (ANOVA and regression)
- By itself only produces a single best estimate (No C.I.’s)
- Can use resampling approaches to get C.I.’s
- Many OLS estimators have been duplicated by ML estimators
Parameter Estimation | Ordinary Least Squares (OLS)
R Interlude | Complete Exercises 3.1-3.2
Recall type 1 and type 2 errors
Power | underappreciated aspect of experimental design
- Type 1 error - \(\alpha\) - incorrectly rejecting a true null hypothesis
- This is saying that there is an effect when there isn’t)
- Type 2 error - \(\beta\) - incorrectly accepting a false null hypothesis
- This is saying that there isn’t an effect when there is)
- Power is the probability of rejecting a false null hypothesis
- Mostly we shoot for a power of around 80%
- Power can be calculated post hoc or a priori
Power | the things one needs to know
\[ Power \propto \frac{(ES)(\alpha)(\sqrt n)}{\sigma}\]
Power | what we usually want to know
Power | rough calculation
Linear Models - a note on history
Linear Models - a note on history
Bivariate normality
Covariance and correlation
Anscombe’s Quartet
Anscombe’s Quartet
Mean of x in each case 9 (exact)
Variance of x in each case 11 (exact)
Mean of y in each case 7.50 (to 2 decimal places)
Variance of y in each case 4.122 or 4.127 (to 3 decimal places)
Correlation between x and y in each case 0.816 (to 3 decimal places)
Linear regression line in each case \[ y = 3.00 + 0.50x\]
A linear model to relate two variables
Many approaches are linear models
- Is flexible: Applicable to many different study designs
- Provides a common set of tools (lm in R for fixed effects)
- Includes tools to estimate parameters:
- (e.g. sizes of effects, like the slope, or change in means)
- Is easier to work with, especially with multiple variables
Many approaches are linear models
- Linear regression
- Single factor ANOVA
- Analysis of covariance
- Multiple regression
- Multi-factor ANOVA
- Repeated-measures ANOVA
Plethora of linear models
General Linear Model (GLM) - two or more continuous variables
General Linear Mixed Model (GLMM) - a continuous response variable with a mix of continuous and categorical predictor variables
Generalized Linear Model - a GLMM that doesn’t assume normality of the response (we’ll get to this later)
Generalized Additive Model (GAM) - a model that doesn’t assume linearity (we won’t get to this later)
Linear models
All an be written in the form
response variable = intercept + (explanatory_variables) + random_error
in the general form:
\[ Y=\beta_0 +\beta_1*X_1 + \beta_2*X_2 +... + error\]
where \(\beta_0, \beta_1, \beta_2, ....\) are the parameters of the linear model
linear model parameters
linear model parameters
linear models in R
All of these will include the intercept
All of these will exclude the intercept
Need to fit the model and then ‘read’ the output
Model fitting and hypothesis tests in regression
\[H_0 : \beta_0 = 0\] \[H_0 : \beta_1 = 0\]
full model - \(y_i = \beta_0 + \beta_1*x_i + error_i\)
reduced model - \(y_i = \beta_0 + 0*x_i + error_i\)
- fits a “reduced” model without slope term (H0)
- fits the “full” model with slope term added back
- compares fit of full and reduced models using an F test
Model fitting and hypothesis tests in regression
Hypothesis tests in linear regression
Estimation of the variation that is explained by the model (SS_model)
SS_model = SS_total(reduced model) - SS_residual(full model)
The variation that is unexplained by the model (SS_residual)
SS_residual(full model)
Hypothesis tests in linear regression
Hypothesis tests in linear regression
\(r^2\) as a measure of model fit
\[r^2 = SS_{regression}/SS_{total} = 1 - (SS_{residual}/SS_{total})\] or \[r^2 = 1 - (SS_{residual(full)}/SS_{total(reduced)})\] Which is the proportion of the variance in Y that is explained by X
Relationship of correlation and regression
\[\beta_{YX}=\rho_{YX}*\sigma_Y/\sigma_X\] \[b_{YX} = r_{YX}*S_Y/S_X\]
Residual Analysis | did we meet our assumptions?
- Independent errors (residuals)
- Equal variance of residuals in all groups
- Normally-distributed residuals
- Robustness to departures from these assumptions is improved when sample size is large and design is balanced
Residual Analysis | did we meet our assumptions?
\[y_i = \beta_0 + \beta_1 * x_I + \epsilon_i\]
Residual Analysis
Residual Analysis
Handling violations of the assumptions of linear models
What if your residuals aren’t normal because of outliers?
Nonparametric methods exist, but these don’t provide parameter estimates with CIs.
Robust regression (rlm)
Randomization tests
Anscombe’s quartet again | what would residual plots look like for these?
Anscombe’s quartet again | what would residual plots look like for these?
Residual Plots | Spotting assumption violations
Residuals | leverage and influence
- 1 is an outlier for both Y and X
- 2 is not an outlier for either Y or X but has a high residual
- 3 is an outlier in just X - and thus a high residual - and therefore has high influence as measured by Cook’s D
Residuals | leverage and influence
Leverage - a measure of how much of an outlier each point is in x-space (on x-axis) and thus only applies to the predictor variable. (Values > 2*(2/n) for simple regression are cause for concern)
Residuals - As the residuals are the differences between the observed and predicted values along a vertical plane, they provide a measure of how much of an outlier each point is in y-space (on y-axis). The patterns of residuals against predicted y values (residual plot) are also useful diagnostic tools for investigating linearity and homogeneity of variance assumptions
Cook’s D statistic is a measure of the influence of each point on the fitted model (estimated slope) and incorporates both leverage and residuals. Values ≥ 1 (or even approaching 1) correspond to highly influential observations.
R INTERLUDE | Complete Exercises 3.5-3.6
How to present your statistical results
Style of a results section
- Write the text of the Results section concisely and objectively.
- The passive voice will likely dominate here, but use the active voice as much as possible.
- Use the past tense.
- Avoid repetitive paragraph structures. Do not interpret the data here.
Function of a results section
The function is to objectively present your key results, without interpretation, in an orderly and logical sequence using both text and illustrative materials (Tables and Figures).
The results section always begins with text, reporting the key results and referring to figures and tables as you proceed.
The text of the Results section should be crafted to follow this sequence and highlight the evidence needed to answer the questions/hypotheses you investigated.
Important negative results should be reported, too. Authors usually write the text of the results section based upon the sequence of Tables and Figures.
Summaries of the statistical analyses
May appear either in the text (usually parenthetically) or in the relevant Tables or Figures (in the legend or as footnotes to the Table or Figure). Each Table and Figure must be referenced in the text portion of the results, and you must tell the reader what the key result(s) is that each Table or Figure conveys.
- Tables and Figures are assigned numbers separately and in the sequence that you will refer to them from the text.
- The first Table you refer to is Table 1, the next Table 2 and so forth.
- Similarly, the first Figure is Figure 1, the next Figure 2, etc.
- Each Table or Figure must include a brief description of the results being presented and other necessary information in a legend.
- Table legends go above the Table; tables are read from top to bottom.
- Figure legends go below the figure; figures are usually viewed from bottom to top.
- When referring to a Figure from the text, “Figure” is abbreviated as Fig.,e.g., (Fig. 1. Table is never abbreviated, e.g., Table 1.
Example
For example, suppose you asked the question, “Is the average height of male students the same as female students in a pool of randomly selected Biology majors?” You would first collect height data from large random samples of male and female students. You would then calculate the descriptive statistics for those samples (mean, SD, n, range, etc) and plot these numbers. Suppose you found that male Biology majors are, on average, 12.5 cm taller than female majors; this is the answer to the question. Notice that the outcome of a statistical analysis is not a key result, but rather an analytical tool that helps us understand what is our key result.
Differences, directionality, and magnitude
Report your results so as to provide as much information as possible to the reader about the nature of differences or relationships.
For example, if you are testing for differences among groups, and you find a significant difference, it is not sufficient to simply report that “groups A and B were significantly different”. How are they different? How much are they different?
It is much more informative to say something like, “Group A individuals were 23% larger than those in Group B”, or, “Group B pups gained weight at twice the rate of Group A pups.”
Report the direction of differences (greater, larger, smaller, etc) and the magnitude of differences (% difference, how many times, etc.) whenever possible.
Statistical results in text
Statistical test summaries (test name, p-value) are usually reported parenthetically in conjunction with the biological results they support. This parenthetical reference should include the statistical test used, the value, degrees of freedom and the level of significance.
For example, if you found that the mean height of male Biology majors was significantly larger than that of female Biology majors, you might report this result (in blue) and your statistical conclusion (shown in red) as follows:
- “Males (180.5 ± 5.1 cm; n=34) averaged 12.5 cm taller than females (168 ± 7.6 cm; n=34) in the pool of Biology majors (two-sample t-test, t = 5.78, 33 d.f., p < 0.001).”
If the summary statistics are shown in a figure, the sentence above need not report them specifically, but must include a reference to the figure where they may be seen:
- “Males averaged 12.5 cm taller than females in the pool of Biology majors (two-sample t-test, t = 5.78, 33 d.f., p < 0.001; Fig. 1).”
Statistical results in text
- Always enter the appropriate units when reporting data or summary statistics.
- for an individual value you would write, “the mean length was 10 cm”, or, “the maximum time was 140 min.”
- When including a measure of variability, place the unit after the error value, e.g., “…was 10 ± 2.3 m”.
- Likewise place the unit after the last in a series of numbers all having the same unit. For example: “lengths of 5, 10, 15, and 20 m”, or “no differences were observed after 2, 4, 6, or 8 min. of incubation”.
